Delivering information about an image corresponding to an object at a particular location

ABSTRACT

An information delivery system has a computational device connected over a network with a server and associated storage device. The computational device is configured with functionality that generates a message requesting information relating to a particular geolocation that is stored in association with the server. The server identifies one or more files corresponding to the location information in the message and delivers them to the computational device, which compares information in the files with a visual image relating to an object selected by a computational device user and with an object type relating to the visual image, and displays information in a file if the visual image selected by the user matches visual image information in the file.

1. FIELD OF THE INVENTION

The present disclosure relates to the delivery of information about anobject that is proximate to a known location based upon an image andgeolocation of the object.

2. BACKGROUND

With the vast quantity of information that is now stored on networkdevices, and which is available for consumption over networks such asthe Internet, information retrieval systems have been developed tofacilitate the rapid and intuitive retrieval of this information.Requests or queries for information stored on these networks can beentered into an information retrieval system, and the system can operateto identify and to return a listing of information or network pages thatmost closely match the query. Different types of information retrievalsystems have been developed to accommodate the identification anddelivery of different types of information in different ways. Systemshave been developed which operate to retrieve information based on keywords or phrases in a search request or query, or to retrieveinformation that is germane to a current location of a mobile devicebased on geolocation or proximity information provided by the mobiledevice. Another type of system for retrieving network information is acontext-based image retrieval (CBIR) system, which operates to retrieveinformation based upon requests that include image information.

3. BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing elements comprising an information retrievalsystem connected to a public network.

FIG. 2 is a diagram showing functional blocks comprising a mobile ornon-mobile communication device connected to the public network havingcontent retrieval functionality.

FIG. 3 is a diagram showing a plurality of location defined informationfiles maintained in a storage means.

FIG. 4 is a diagram showing the organization of and informationmaintained in an instance of a location defined file.

FIGS. 5A and 5B are diagrams showing a logical process for identifyingand delivering information about an object located at a particulargeolocation.

4. DETAILED DESCRIPTION

The degree to which an information retrieval system is useful oftendepends on the ability of the system to easily, quickly and accuratelyretrieve information of interest to a requestor, without returninginformation that is of marginal or no interest. To this end, informationretrieval systems have been developed that rank or score the informationthat is identified during a search, and this information can bepresented in a search engine results page (SERP) in rank order with theclosest match first. A typical search can return a large number ofresults, which can take a long time to review and identify informationthat is interest to the individual requesting the information.Furthermore, queries based on image information can be particularlylengthy and imprecise, as only a limited amount of information canreasonable be stored about any particular image (fingerprint informationfor example), and so, finding a match to an image of interest in a querycan take a relatively long time and return a large number of items/pagesfor display on a SERP.

In lieu of the forgoing, it would be advantageous if the operation of aninformation retrieval system is restricted such that it can onlyidentify and deliver information that relates to a particular objectlocated at a particular geographic position. I have designed an imageand location-based information retrieval system that can identify andonly deliver information that specifically relates to a particularobject, such as an object of art, a person, a place or anything that isidentifiable based upon visual image and geographic position informationassociated with the object. This retrieval system can access speciallocation defined files maintained in a network that can have geographiclocation information about the object, classification information aboutthe type of the object, visual image information that relates to theobject and descriptive information about the object.

According to one embodiment, the retrieval system operates, under usercontrol, to generate a search query having a current location of acommunication device or location information entered by a user of thecommunication device and having information relating to a type of theobject or object classification. The retrieval system can process thesearch query and return information that is limited to the type of theobject at the location in the search query. The information retrieved bythe system can be in the form of a location defined file that has visualimage information relating to the object that is the subject of thesearch query. The retrieval system can then compare the visual imageinformation in the retrieved location defined file with visual imageinformation residing on the communication device which relates to theobject of interest to the communication device user, and if the visualinformation matches, or is similar, the retrieval system can display thelocation defined file on the communication device to be viewed by thedevice user. In this manner, only information of particular interest tothe device user is displayed for viewing, and the information retrievalprocess is accelerated.

According to another embodiment, the retrieval system operates togenerate a search query having a current location of the user, or alocation of the object entered by the user, having an object typeinformation, and having visual image information relating to the objectof interest. The retrieval system processes the search query to identifylocation defined files maintained at one or more locations on a networkdevice, it examines the identified files looking for a match between theobject type information in the search query and the object typeinformation in the location defined files, it then matches the visualimage information in the search query with the visual image informationin the location defined files, and only delivers information in alocation defined file that meets all the criteria included in the searchquery.

Alternatively, more than one search query can be generated forprocessing by the retrieval system. For instance, a first query can begenerated that only has location information, and the retrieval systemcan use the location information to identify a set of location definedfiles. Then a second message can be generated having the object typeinformation, or have the object type and visual image information, whichthe system can use to compare against the set of location defined filesalready identified. As will be described later, depending upon thecapabilities of a user communication device, and depending upon how theretrieval system is configured, the process of identifying locationdefined files (and information in the files) for display can occur in anumber of different network locations. An information retrieval system,such as the one described above, operates to only deliver informationabout an object that is of interest to the user, thereby saving the usera significant amount of time sifting through information that is not ofinterest or that has no relationship to the object of interest to theuser. The special location defined files can be maintained in any typeof mass storage device associated with a computational device that isconnected to and accessible over the network.

This information retrieval system does not have to search through allvisual image information relating to objects available on a networklooking for a match to the visual image information in the query, butrather only searches for files having location tags that correspond tothe location of an object relating to the image of interest, and thensearches through these location defined files looking for a match to theimage type in the query, and alternatively to the visual imageinformation also, and only delivers information about the objectcorresponding to the object type, or to the object type and visual imageinformation in the query. In this regard, FIG. 1 shows such aninformation retrieval system 100, elements of which can all be connectedto a public network 130 such as the Internet, or connected to somecombination of public and private networks. As shown in FIG. 1, thesystem 100 has two communication devices 101 and 105, each of which areconnected to the network 130 by wired or wireless links 102 and 106respectively. The device 101 can be a stationary, or semi-stationarycomputational device such as a desk-top or lap-top computer, and thedevice 105 can be a mobile communication device, such as a smart phoneor similar device. FIG. 1 also shows computational devices 110A, 110B,and 110C being connected, via links 115, 116 and 117 respectively, tothe network 130 with associated storage means 111A, 111B, and 111C. Thecomputational devices can be network servers or any type ofcomputational device able to support the associated storage means, andthe storage means can be physical or virtual disk storage, volatile ornon-volatile semi-conductor memory, or any type of mass storage that canbe addressed by the computational devices. Some or all of the storagemeans can be separate from the computational devices 110A-110C, or itcan be tightly integrated in the servers. A detailed description of thefunctional elements comprising the computational devices, and theassociated mass storage means will be included later.

Generally, the communication devices 101 and 105 can be configured withfunctionality that operates to capture image information (i.e., networksearch engine functionality), from the environment proximate to thedevices or to receive image information via the search engine and storedon the communication device, to classify the image information as anobject type (i.e., a work of art such as a picture type object, asculpture type object, a tapestry type object, an image of a person typeobject, etc.), and which operates to receive geographic positioninformation relating to the image of an object of interest to the deviceuser. The communication devices can also operate to pass this imageinformation, object type information, and geographic positioninformation to an information retrieval function or application, runningon the communication devices, which operates in conjunction with asearch engine to generate a search query that it uses to retrieveinformation about the image stored on the network, and that is locatedat the geographic position included in the query, that is of the sameobject type as in the query, and that is compared to be the same orsimilar image as the one in the query.

With continued reference to FIG. 1, the computational devices 110,hereinafter referred to as network servers 110, generally operate toreceive a search query from the communication devices 101 and 105, andto use the information in the query to identify information maintainedin the associated storage means, which can be very large databasesmaintained by any organization that provides client network searchengine functionality implemented on the communication devices 101 and105. The network servers also operate to return the information,identified as the result of the network service processing the searchquery, to the search engine functionality running on the communicationdevices. The communications device 105 will now be described below withreference to FIG. 2.

While the communication device 105 is illustrated in FIG. 1 as a mobiledevice, other types of communication devices can be configured with thesame or similar functionality as that shown in FIG. 2. Tablet computers,laptop or desktop, or any type of communication device capable ofconnecting to a network can be configured to have similar functionality.The communication device 105 in FIG. 2 is configured to have a networkbrowser or search engine 205 and a network interface in block 200, alocation-based information retrieval functionality 210, geographiclocation service functionality 220, computer vision and imageidentification and classification functionality 230, and imageacquisition and storage functionality 240.

The browser or search engine can be any one of a number of commerciallyavailable applications running as a client on the communication devicethat operate to accept search terms from a device user, to generatesearch requests or queries that are transmitted over the networkinterface to a service provider associated with the search engineapplication, and then returns or delivers search results to theapplication running on the user device for display in a SERP. Thelocation-based information retrieval function 210 is specially designedto receive position information from the GPS/IPS functionality 220, toreceive image information and object classification information(corresponding to the image) from the computer vision and imageclassification function 230, and to pass this position and image andobject classification information to the network search engine where theinformation is used to generate a query. Alternatively, the imageinformation may not be sent to the retrieval function 210, in which casethe search engine can return some number of location defined fileshaving object image information that is compared (using the computervision function 230) to the image information in the store 240. Whilethe retrieval function 210 is shown as being separate from the searchengine 205, this does not have to be the case, as it can alternativelybe tightly integrated into the search engine function. The operation ofthe GPS function is well known and so will not be described in detailhere other than to indicate that this functionality typically runs as anative application on the communication device and periodicallydetermines the current position (geographic or indoor position) of thecommunication device.

The image acquisition and store function 240 can be a camera, imagescanner, or any type of device that is capable of capturing atwo-dimensional or three-dimensional image from the environment that isproximate to the communication device. The function acquisition functionalso operates to store images retrieved by the search engine 205.Regardless of what type of device is employed to acquire the images,these images have information that corresponds to some type of object,such as a work of art (i.e., painting, sculpture, tapestry, pottery,etc.), a human face, a building, a landscape feature, or any object forwhich image information can be captured in some manner. The imageinformation maintained by the acquisition function is accessible by thecomputer vision and image classification function 230, and used byfunction 230 to detect or extract feature information from the image,determine which of the feature information to use during an objectclassification process, and then identifying an object class that mostclosely matches the image features. For example, after an image iscaptured, the function 130 can operate to identify edges, lines, areasof color and contrast, and other features associated with the capturedimage. Each of these features are represented as visual words, and animage feature histogram can be generated that is used to identify anobject class. A determination that feature information in a histogramcorresponds to an image class is typically a learned process. Morespecifically, a determination that an image belongs to an object classis based on a dictionary or code book comprising a plurality of imagefeatures, and each feature is described using visual words. An instanceof a bag of visual words (BoW) can equate to an object class.

As described previously, the retrieval function 210 operates to receiveposition information, image information and object classificationinformation, and to pass this information along to the search engine 205which generates a search query having this information, and transmitsthe query over the network 130 to one or more servers, 110A, 110B or110C, that the search engine has access to. The server or servers thenoperate on the information in the query to identify one or more locationdefined files corresponding to at least the location information, andalternatively to identify files corresponding to the location, andobject type and visual image information in the query.

FIG. 3 is a diagram of any one of the information storage means 111A,111B and 111C, showing a plurality of location defined files, files 300to file.N (with N being an integer number), each one of which can haveone or more sub-files (file 310 to file.N comprising file 300, and files320 to file.N, with one or more of the sub-files having informationcorresponding to an object of interest. The information comprising eachfile can be metadata type information that is gleaned by a web crawlerfrom web pages published on the network 130. This metadata can becomprised of object location information, information that isdescriptive of the object and visual image information relating to theobject, such as visual fingerprint information.

FIG. 4 shows a format that can be used to store information in alocation defined file, such as the file 310. This file is shown as beingcomprised of location information 410, object class type information420, image information 430, and object information 440, but it should beunderstood that the file 310 can have other information as well. Thelocation information can be a tag or an index that corresponds to aknown, geographic location of an object for which information is beingrequested. Only files having a location tag that is the same as thelocation information in a query (whether the query location informationis the current location of a user device or the query location ismanually entered by a device user) will be identified by the searchengine during a search. Limiting the scope of a search in this manner,greatly increases the probability that only information relating to anobject of interest will be returned by the search engine, and reducesthe time it takes to perform the search. The object classificationinformation 420 can be a short descriptor of the type of objectassociated with the image. This descriptor can be, Picture, Sculpture,Building, Person Face, Landscape image, or any type of object class thatcan be described with a few words. This object classificationinformation can also include sub-class information, such as the subjectmatter of a painting (i.e., Portrait Painting, Landscape Painting, StillLife Painting, etc.). The image information 430 can be any type ofvisual record of the image of interest. I can be a record havingdigitized pixel information, or it can be a fingerprint of the visualrecord against which the image in the query is compared.

The object information 440 in FIG. 4 is comprised of information thatrelates, in some manner, to the object identified as the result of thesearch. In the case that the object is a work or art, this informationcan be the name and biographical information about an artist thatcreated the object, it can be descriptive information about the subjectmatter of the object, which in the case of a painting can be thegeographic position corresponding to the subject matter comprising thepainting (i.e., particular bridge over a particular body of water). Thesearch query can be tailored by the device user so that all of theobject information 440 associated with the object of interest isreturned, or only a portion of this object information is returned bythe search engine.

Typically, the web pages from which a web crawler extracts metadata tocreate the location defined files 310, are created by an organizationthat has some interest in disseminating information about an object,such as a work of art, that could lead to individuals attending aparticular venue or activity. In this regard, these organizations (i.e.,art museum, sporting event, concert event, etc.) can create a web pagehaving the location information associated with their page that can berecognized and copied by a web crawler and entered into the informationfile 310 format in the form of a location tag. As described earlier, theinformation comprising a location tag can be geolocation informationdetermined by a positioning service application or by a socialnetworking application running on a device, or it can be locationinformation that is manually entered into the device. In the later case,the location information does not need to be geolocation information,but can be street address information, ZIP code information, the name ofa city, or any information that uniquely identifies a location of anobject. The process by which the system 100 operates to process a searchquery and return results is now described with reference to FIG. 5.

FIG. 5 is a flow diagram showing a logical process by which the system100 can operate to retrieve information corresponding to a particularobject for an individual requesting the information. As previouslydescribed, a request for information about an object is generated in theform of a search query that may or may not include a visual image orobject type information corresponding to the object of interest in thequery. In the event that visual image and object type information isincluded in the query, the system 100 operates to only retrieveinformation in a location defined file that has matching image, locationand object type information. Alternatively, in the case that a query isgenerated that does not have visual image and/or object typeinformation, the system 100 can retrieve some number of the locationdefined files stored in association with the network servers that havelocation information that matches that in the search query. Theselocation defined files can have image fingerprint information, or someother type of compressed or abridge image information that a computervision application running on the requestor device can process todetermine which of the files have matching image information, and thenonly information comprising these location defined files is displayed tobe viewed on the user/requestor device. FIG. 5 describes the operationof the system 100 to process search queries having visual image andobject type information and search queries not having visual imageand/or object type information, and the system 100 can be configured toprocess either or both types of search queries.

Referring now to FIG. 5A, after the system 100 is initialized, andsubsequent to a user capturing or identifying a visual image about whichthey are interested to learn, the user at 500 interacts with thelocation based content retrieval function 210 running on theircommunication device t (i.e., device 101 or 105 in FIG. 1) to generate asearch query. As described above, this search query can have objectclassification/type information, it may or may not have object locationinformation, and it may or may not have visual image information in someformat. The search engine uses the information in the query to find alocation defined file having information that matches that in the query.At 505 one or more of the serves 110 receive the query and determinewhether it only has location information or not. If only locationinformation is included, then the process proceeds to 510, otherwise theprocess proceeds to 540. In the event that the query only has locationinformation, then at 510 the server identifies all files having locationinformation that is the same as the location information in the query.If at 515 the server identifies any location defined files having alocation tag matching the location information in the query, then at 520the server delivers these files to the use device, otherwise, theprocess terminates. At 525, computer vision and classificationfunctionality 230 running on the user device operates to compare visualimage information, selected by the user and stored on the user device,and to compare object type information relating to the selected visualimage with visual image information in all of the location defined filesreturned in 520, and at 530, any matches can be displayed by the userdevice to be viewed by the user.

Returning to 505 in FIG. 5A, the process proceeds to 540 if the serverdetermines that the query generated in 500 does not have only locationinformation, and if at 540 the server determines that the query has onlylocation and object type information, then at 545 the server examinesall location defined files looking for location tag and objectinformation that matches that information in the query. Alternatively,if at 540 the server determines that the query does not only havelocation and object type information, then the process proceeds to 541where a determination is made that the query has location, object andvisual image information or not, and if so the process proceeds to 542where the server examines all the location defined files for matchinginformation, and at 543 identifies all matching files, otherwise theprocess returns to 505. If at 550 and 543 the server identifies fileswith matching information matching the queries, then the server at 555responds by sending the identified files to the user device, otherwise,the process terminates.

At 560 in FIG. 5B, the computer vision functionality running on the userdevice compares visual image information, relating to an object selectedby the user, with visual image information in each one of the filesidentified at 550 and sent to the user device at 555, and if at 565 itis determined that matching information is found in any of the files,then at 570 at least some of the information comprising those files withmatching information is displayed on the user device to be viewed.

What is claimed is:
 1. Method of identifying and delivering informationcorresponding to an image of a particular object of interest,comprising: creating and maintaining on a first computational deviceconnected to a network a plurality of location defined files, each ofthe location defined files having unique geographic position informationcorresponding to a location of a particular object, visual imageinformation relating to the particular object, object type information,and having information that is descriptive of the particular object;selecting, by a user of a second computational device, a visual image ofa particular object of interest located at a particular geographicposition and that is stored on the second computational device, andentering the visual image of the particular object of interest into acontent retrieval application running on the second computationaldevice; classifying, by the content retrieval application, the selectedvisual image of interest as an object type, and receiving from thesecond computational device the geographic position information relatingto the location of the particular selected object of interest;generating, by the content retrieval application, a message comprising arequest for all location defined files corresponding to the particulargeographic position of the selected object of interest and sending themessage to the first computational device; the first computationaldevice receiving the message and using the geographic positioninformation to identify one or more location defined files, andresponding to the request message by sending over the network to thesecond computational device at least some information comprising theidentified one or more location defined files; receiving at the secondcomputational device the one or more location defined files, anddetermining that at least one of the received files has visual imageinformation and object type information that matches the user selectedvisual image of the particular object of interest and associated objecttype classification information; and displaying by the secondcomputational device at least some of the information in the locationdefined file on the second computational device for the user to view. 2.The method of claim 1, wherein the selected visual image comprises animage of the particular object of interest captured from the environmentthat is proximate to the second computational device or is an image ofthe particular object of interest copied from a web page of a networksite.
 3. The method of claim 2, wherein the environment comprises anindoor or an outdoor location.
 4. The method of claim 1, wherein theparticular object of interest comprises an object of art, a person, or aplace.
 5. The method of claim 1, wherein information descriptive of theparticular object of interest comprising each of the location definedfiles is any one or more of a name of a creator, creator title,description of the particular object of interest, multimedia contentrelating to the particular object of interest, price of the particularobject of interest, origin of the particular object of interest,materials used to create the particular object of interest, locationassociated with the particular object of interest, and identificationnumber of the particular object of interest.
 6. The method of claim 1,wherein the content retrieval application comprises computer visionfunctionality operating in cooperation with a network browser or searchengine.
 7. Method for receiving information corresponding to aparticular object of interest, comprising: selecting, by a user of afirst computational device, a visual image of a particular object ofinterest that is stored on the first computational device and located ata particular geographic position, and the user entering the visual imageof the particular object of interest into a content retrievalapplication running on the first computational device; classifying, bythe content retrieval application, the selected visual image as anobject type, and the content retrieval application receiving from ageographic position system running on the first computational device orfrom the user, geographic position information relating to theparticular location of the particular object of interest; generating, bythe content retrieval application, and sending a message over a networkto a second computational device requesting all location definedcomputer files maintained by the second computational device that haveinformation about the particular object of interest located at theparticular geographic position; receiving, over the network from thesecond computational device in response to the request message, one ormore location defined computer files corresponding to the particulargeographic location, each of the location defined computer files havingvisual image information relating to the particular object of interestand having information that is descriptive of the particular object ofinterest; determining by the first computational device, that at leastone of the received location defined files has visual image informationthat matches the particular user selected object of interest, and hasobject type information that corresponds to the particular user selectedobject of interest; and displaying by the first computational device atleast some of the information in the matching location defined computerfile.
 8. The method of claim 7, wherein the selected visual image of theparticular object of interest comprises an image captured from theenvironment that is proximate to the first computational device or is animage of the particular object of interest copied from a web page of anetwork site.
 9. The method of claim 8, wherein the environmentcomprises an indoor or an outdoor location.
 10. The method of claim 7,wherein the particular object of interest comprises an object of art, aperson, or a place.
 11. The method of claim 7, wherein informationdescriptive of the particular object of interest comprising the locationdefined computer files is any one or more of a name of a creator,creator title, description of the particular object, multimedia contentrelating to the particular object of interest, price of the particularobject of interest, origin of the particular object of interest,materials used to create the particular object of interest, locationassociated with the particular object of interest, and identificationnumber of the particular object of interest.
 12. The method of claim 7,wherein the content retrieval application comprises computer visionfunctionality operating in cooperation with a network browser or searchengine.